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HANDWRITTEN AND VOICE CONTROL OF VEHICLE COMPONENTS 

FIELD OF THE INVENTION 

The present invention relates to handwriting and voice recognition systems in 
general, and more particularly to handwriting and voice recognition systems for use in a 
5 vehicle. 

BACKGROUND OF THE INVENTION 

Voice recognition systems have been utilized in a variety of applications to 
command electronic devices. Recent attempts have been made to employ such voice 

10 recognition systems to allow drivers to control automobile appliances, such as 
vehicle-based cellular telephones or electrical windows. Such applications make it 
possible for drivers to keep their eyes on the road instead of on the appliances while 
driving, thereby enabling drivers to drive more safely. However, such systems are 
adversely affected by the ambient noise present in environments such as the interior of a 

1 5 vehicle, particularly when the windows are open while driving at high speeds. Thus, the 
need exists for vehicle appliance control that allows a driver to keep his eyes on the road 
while not being affected by ambient vehicle noise. 

SUMMARY OF THE INVENTION 

20 It is an object of the present invention that a driver may operate automobile 

appliances safely while driving. A system is provided whereby the driver's eyes need not 
leave the road, and the driver's hands need not leave the steering wheel or gearshift, 
when operating an automobile appliance. 
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It is a further object of the present invention to simplify the operation of vehicle 
appliances using an improved command recognition system. 

It is yet another object of the present invention to provide a command recognition 
system capable of use in noisy environments. 
5 It is another object of the present invention to simplify command recognition 

operation when used by different users. 

It is an advantage of the present invention to combine handwriting and voice 
recognition to eliminate the need for speaker independent voice recognition technology 
which today is hard to implement in a noisy environment 
10 It is an additional advantage of the present invention that handwriting recognition 

for providing commands is not influenced by the automobile's noisy environment 

It is yet another advantage of the present invention to provide command 
recognition operation independent of noise generated by other persons in the vehicle. 

It is a feature of the present invention that devices may be operated by both voice 
15 and handwriting commands to increase the available instruction set for sophisticated 
appliances such as personal computers or cellular telephones when sending short text 
messages. 

It is an additional feature of the present invention to offer combined handwriting 
and speaker dependent recognition in order to achieve high recognition credibility 
20 desirable for work in a vehicle environment. 

It is an additional feature of the present invention that double validation of a user 
identity is employed using both the voice print and the signature characteristics of the 
user. 
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The present invention is embodied in an improved user interface device for 
controlling at least one appliance within a vehicle comprising a command recognition 
module having a command data set. The command recognition module is adapted to 
receive handwriting and voice signals, to associate said handwriting and voice signals 
5 with the command data set such that at least one command is used by said at least one 
appliance. The command recognition module is further adapted to generate a command 
signal sequence upon associating at least one of the handwriting and\or voice signals 
with the at least one command. A main module operatively connects to the command 
recognition module and operates to relay the command signal to the at least one 

1 0 appliance. An appliance interface connects between the at least one appliance and the 
main module and is adapted to relay the command signal to the appliance. 

The command recognition module further includes a microphone and a voice 
recognizer operatively connected to the microphone to receive voice signals and to 
generate a command signal upon associating at least one of the voice signals with the at 

1 5 least one command. Furthermore, a touchpad and a handwriting recognizer operatively 
connected to said touchpad is included to receive handwritten signals and to generate a 
command signal upon associating at least one of said handwritten signals with said at 
least one command. Handwritten signals may include handwritten characters, words, 
numbers, scribbles, or any gesture that a user can perform with his hands or fingers, 

20 including three dimensional gestures, thus the touchpad can be replaced by any kind of 
hand-or-finger responsive input device like a joystick, a three dimensional mouse, etc., 
or any other known body-part-responsive input device adaptable to a human 
communication method, such as a head movement. A speaker is connected to either the 
command recognition or the main module to give feedback to the driver. The feedback 
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may be in the form of a playback of the recognized command before its execution and/or 
talking menus using recorded speech or synthetic speech such as known text-to-speech 
(TTS) technology. A transparent display mounted on the front window may also be used 
to generate a visual feedback. Alternatively, after a character or digit is entered and/or 
5 after an entry sequence is completed (e.g., word, telephone number) the entry may be 
played back or displayed. Other features and advantages of this invention will be made 
apparent upon review of the drawings and detailed description of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be understood and appreciated more fully from the 
following detailed description taken in conjunction with the appended drawings in 
which: 

Figs. 1 and 2 are functional block diagrams of a voice and handwriting recognition 
system constructed and operative in accordance with a preferred embodiment of the 
present invention; and 

Fig. 3 is a functional block diagram of a dual validation mode voice and 
handwriting recognition system constructed and operative in accordance with a preferred 
embodiment of the present invention. 

20 DETAILED DESCRIPTION OF THE PRESENT INVENTION 

With reference to the drawings for purposes of illustration, the present invention is 
embodied in a command recognition device 20 (Fig. 1) that enables the operation of 
electronic appliances in a vehicle or the like, represented conceptually by a steering 
wheel 18, by using handwritten commands through the use of handwriting recognition 
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technology known in the art. Known voice recognition technology may additionally be 
combined with handwriting recognition technology to provide a reliable and robust 
method of interacting with such electronic appliances, systems or devices. Electronic 
appliances 22 may include, but are not limited to, a cellular telephone 24, a Global 
5 Positioning System (GPS) navigation system 26, a personal computer (PC) 28, a radio 
30, an alarm system 32, electric windows (not shown), a sunroof (not shown), and the 
like. 

With reference to Fig. 2, appliances 22 connect to the device 20 through an 
appliance interface 34. The appliance interface 34 is preferably adapted for use with one 

10 or more appliances or types of appliances and may be adapted to conform with appliance 
hardware standards and\or software protocols for remote operation thereof. The interface 
module 34, implemented in hardware and/or software, is the physical module with which 
appliances connect via electrical connectors 36. Interface 34 is controlled by a main 
program module 35 which selects the commands to be sent to each appliance. Interface 

1 5 module 34 also produces the correct voltage and current needed to communicate with 
the appliances. For example, if the user wishes to communicate with a PC via the RS232 
protocol, then translation of digital commands produced by the main program module 35 
into actual +/-12 volt pulses (as required by the RS232 protocol) will be performed by 
the interface module 34. Command sets corresponding to each of the appliances is 

20 maintained by the main program module 35. An example of such interface adaptability 
is found in conventional universal remote control devices for use with entertainment 
systems where the remote control device may be trained by the user to cooperate with 
appliances and/or includes a library of known appliances and specific commands 
corresponding to such appliances. In the latter instance the user selects the library of 
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commands that corresponds to the appliances. The connection leads 36 between the 
appliance and the interface may vary and include, but are not limited to, wire, infrared, 
serial, or bus connections. In an alternative embodiment, the appliances may be 
assembled with the interface module 34 and may include any number of conventional 
5 interface configurations. 

Device 20 preferably includes a voice recognizer 38 connected to a microphone 42 
and a handwriting recognizer 44 connected to writing pad 46. Command recognition as 
used herein refers to handwritten or spoken command signals being recognized by a 
handwriting or voice recognizer engine and then matched with preprogrammed 
10 commands contained in a command data set which are associated with the command 
signal. 

The voice recognizer 38 receives voice commands through the microphone 42 and 
searches for a matching voice command within a known list of voice commands tailored 
for the appliances to which device 20 is connected. The microphone may be placed in a 

1 5 vehicle console, sun visor, or in any location suitable for the convenient reception of 
voice commands from the driver or other passengers. Voice commands may also be 
input via an external microphone, such as via a cellular phone, or otherwise into the 
voice recognizer such as is described in assignee's copending U.S. Patent Application 
No. 09/002,616, the disclosure of which is incorporated herein by reference. It will be 

20 appreciated that such voice commands may be trained by the user and/or provided in a 
pre-trained library of appliance brand specific commands. 

The handwriting recognizer 44 receives symbols provided by the user through 
handwriting pad 46. The handwriting pad 46 is preferably located in a position 
convenient for the driver such as on the steering wheel or on the gearshift. Alternatively, 
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handwriting commands may be input via an external touchpad, such as a laptop f s 
touchpad connected to device 20. A handwriting recognizer of the type suitable for this 
purpose is disclosed in U.S. Patent Nos. 5,659,633 and 5,774,582 and U.S. Patent 
Application Nos. 08/528293, 08/418530, 08/282187, and 08/878741, the disclosures of 
5 which are incorporated herein by reference. It will be appreciated that handwritten 
symbols corresponding to appliance commands may be trained by the user and/or 
provided in a pre-trained library of application specific commands. 

It will be appreciated by those skilled in the art that the command libraries for the 
command recognition module may include a standard set of commands adapted for use 
10 with appliances such as cellular phones or personal computers, such as commands 
relating to voice dialing, number dialing, getting e-mail, operating an alarm system, and 
the like. 

Main program module 35 provides an interface to control the transfer and 
distribution of commands between the appliance interface 34 and the recognition 

15 modules 38 and 44. The device 20 preferably draws power from the car's voltage 50 
through a direct connection, being connected to the lighter outlet or directly to the car 
battery. Device 20 may alternatively operate independently on DC batteries. 

A switch 52 is included to permit switching of the device between an operational 
mode and a set-up/training mode. A function of the main program module 35 is to 

20 combine and coordinate the recognition modules 38 and 44 in order to produce the 
proper command to the chosen appliance. The first command after entering the 
recognition or operation mode should be related to identifying the appliance which the 
user wishes to control. Alternatively, a default appliance or the last appliance used 
before system shutdown may be used. Following appliance selection, the main program 
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module 35 will switch to the chosen appliance, sending commands recognized by the 
voice recognizer 38 and/or the handwriting recognizer 44 to that specific appliance. It is 
the role of the main program module 35 to associate the recognized command given by 
the user and the proper appliance command for the appliance selected. 
5 The voice and handwriting recognizers 38 and 44 search for a matching voice or 

handwriting command in a library of commands. There are preferably separate libraries 
for every appliance. Once an appliance is selected, the main program module 35 
instructs the recognizers to load the proper command libraries, typically one for voice 
and one for handwriting. Alternatively, some libraries are loaded and the recognizers are 

10 notified which libraries should be used during the search for a matching command. 
Subsequently, upon receiving an input signal, the recognizers send an identifier of the 
appliance command that was recognized to the main program module 35. Given this 
identifier and the appliance that has been selected, the main module will interpret the 
command and send it to the selected appliance via the appliance interface 34. To enable 

1 5 global commands, such as pick up the cellular phone, there are also voice/handwriting 
signals that will be recognizable no matter what appliance has been currently selected. 
Representations of these signals will be loaded in every library for use by the 
recognizers, or in a global library, and will have unique identifiers that will not be used 
by any other appliance so that the main program module 35 will always be able to 

20 interpret them. Furthermore, a "double validation mode" is provided where the main 
program module 35 validates that both recognizers returned identifiers corresponding to 
the same command, otherwise the command will be rejected. In order to increase the 
effectiveness of the system in the presence of noise, a tolerance level may be set. For 
example, the voice recognizer 38 could return the identifiers of several best matches 
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instead of just one identifier. Should one of the identifiers correspond to an identifier 
returned from the handwriting recognizer 44, this may be deemed an acceptable 
validation. Additionally, a "conforming mode" is also provided in order to allow two 
commands to be entered simultaneously. For example, when the user says "radio" and 
5 writes "5" the device will turn on the radio and select station 5 immediately. 
Furthermore, a "user verification/authentication mode" is provided based on a 
combination of the user's voice and handwriting characteristics rather than the known 
user verification modes that operate on these characteristics separately. The user may 
receive feedback from device 20 through loudspeaker 40 indicating the recognized 

1 0 command was received and sent to the appliance. 

It is appreciated that the same handwriting gesture or spoken command may have a 
different meaning when being interpreted by the main program module 35 depending on 
the appliance selected. For example, the spoken word "up" might mean "close window" 
if "electric windows" have been selected, but it might also mean "increase the volume" 

1 5 when operating the radio. It is further appreciated that a handwriting gesture or spoken 
command may operate more than one appliance at a time, thus providing a "macro" 
capability where one spoken\handwritten gesture is associated with a series of 
commands. For example, a user might say "Jill's settings," whereupon device 20 adjusts 
the seats, mirrors, and steering wheel, and turns on the radio and tunes in Jill's favorite 

20 station. 

The appliance interface 34, command recognition modules 38 and 44, and main 
program module 35 may operate on any conventional hardware platform having a CPU 
and memory adequate to carry out the recognition functions in real time and capable of 
storing the command libraries. Programming logic may be stored in any type of 
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non-volatile memory, for example, on memory chips such as flash memory or EPROMs. 
Optionally, command libraries may be included on removable memory modules (not 
shown) allowing for user training to occur on a conventional PC or to provide appliance 
specific libraries tailored to specialized needs. The libraries may also be loaded from a 
5 PC to device 20 using communication protocols such as RS232. A CPU and memory 
configuration of the type for this purpose includes the H8/3040 microcontroller, 
commercially available from Hitachi, and AMD29F800 Flash memory, commercially 
available from Advanced Micro Devices. 

In a preferred embodiment, the device 20 has two major operation modes, namely, 

1 0 recognition and training modes. Switching between operation mode and training mode is 
controlled by the switch 52. The method for switching between modes using the switch 
may vary. For example, if the user wishes to enter recognition mode, he may press the 
switch once to enter operation mode, enabling device 20 to accept spoken or written 
commands. If the user wishes to train the device 20 for a new command, he may rapidly 

15 press the switch twice so that the main program may switch to training mode. Other 
global commands may be associated with the switch such as for ending a conversation 
via cellular phone or for closing an application window on the computer. Operation 
Mode 

Device 20 in operation mode receives voice commands through a microphone 42. 
20 The microphone 42 receives the human voice and transforms it into an electrical signal. 
This signal is then amplified and digitized by the voice recognizer 38. After digitization, 
the voice recognizer 38 utilizes processing software in order to process the voice signal 
and perform recognition by searching for a matching voice command in a library which 
resides in memory. Following association of the voice signal with a command from the 
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library, the voice recognizer 38 sends the command identifier to the main program 
software module 35 which then interprets it according to the current input state of device 
20 (e.g., appliance has been selected, waiting for appliance command input) and sends 
the command sequence to the proper appliance(s). The speaker 40 may be used by the 
5 recognition modules 38 and 44 or by the main program module 35 to optionally convert 
the command signal into a computer generated voice signal or other playback signal to 
provide the user with feedback as an indication that the voice signal was received and 
properly understood. 

Handwritten commands are received through writing pad 46, such as a touchpad, 

1 0 that when touched at a certain point transmits the touch point f s X and Y coordinates. In 
the case of a touchpad, the input may be provided with a finger. The handwriting 
recognizer 44 receives the character, gesture, or word written on the touchpad and 
searches for a handwriting command match in a library of pre-trained characters, 
gestures, and words. Once the handwriting recognizer 44 has found a match, it signals 

15 the main program module 35. Preferably, the handwriting recognition library includes 
pre-trained alphanumeric characters, gestures, and symbols so the user does not have to 
train the recognizer to recognize them. However, the user may have the option of 
generating symbols convenient to the user for association with appliance commands. 
Training Mode 

20 Training each recognizer 38 and 44 means associating a spoken command or a 

written symbol\gesture\word\character with a command or command sequence that is 
understood by at least one of the appliances by programming such a relationship into the 
device 20. For example, the gesture "arrow up" could mean "close the window, " or the 
uttering a name could activate the cellular phone to dial that person's telephone number. 
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The communication protocols of the appliances used in the present invention are 
typically pre-programmed with device 20. Where a personal computer or other easily 
adaptable computing device is used, software within the PC may be provided to 
communicate with the present invention. In such instances the number of possible 
5 commands depend upon the type of software and operations used by the user. In order to 
associate a command with an operation, the user might program the commands as part of 
the training process. Where operating appliances such as an alarm or a window, device 
20 must possess the actual signals that operates those appliances. Appliance Examples 
A typical example of the operation of the present invention may be seen with 

1 0 respect to a radio in which the command set library for the radio was pre-trained into 
device 20 or entered by the user during training mode. After selecting recognition mode, 
the user says the word "radio". The main program module 35 now routes all the 
recognized commands to the radio. The user then writes or says the number "5." The 
main program then translates the recognized "5" to a digital command understood by the 

1 5 radio as "jump to station number 5" and sends it via interface 34 to the radio. Feedback 
may be provided for every operation performed by the user. For example, when the user 
says "radio" the loudspeaker says "radio is activated" using TTS technology. When the 
user writes "5" the loudspeaker says "station 5 selected" and optionally the name of the 
station, if available. 

20 When activating or deactivating a vehicle alarm system, the user may train device 

20 to learn both spoken and handwritten passwords, signatures, and other user-specific 
signal characteristics as can be seen with additional reference to Fig. 3. Only after 
recognizing both passwords will the main program module 35 send a command to 
activate or deactivate the alarm system. Another related application provides for 
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appliances such as the seat and mirrors to be adjusted according to the user identification 
using personalized command and settings libraries, thus accommodating any one of 
several drivers of the same car. 

When operating a car navigation system, the user may navigate the system's menus 
5 by voice and then enter names of streets by writing them on the touchpad. 

While the present invention has been described with reference to a few specific 
embodiments, the description is intended to be illustrative of the invention as a whole 
and is not to be construed as limiting the invention to the embodiments shown. It is 
appreciated that various modifications may occur to those skilled in the art that, while 
10 not specifically shown herein, are nevertheless within the true spirit and scope of the 
invention. 
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CLAIMS 


10 


15 


What is claimed is: 

1. Apparatus for controlling at least one appliance within a vehicle, said device 
comprising: 

a command recognition module having a command data set and adapted to: 
receive a handwritten signal, 

associate said handwritten signal with at least one command in said command data 
set for controlling said at least one appliance, and 

communicate a command signal associated with said command to said appliance. 

2. The apparatus of claim 1 wherein said command recognition module includes a 
microphone and a voice recognizer operatively connected to said microphone to: 

receive a voice signal, and 

associate said voice signal with said at least one command in said command data 
set for controlling said at least one appliance. 

3. The apparatus of claim 1 wherein said command recognition module includes a 
touchpad and a handwriting recognizer operatively connected to said touchpad to receive 
said handwritten signal. 

4. The apparatus of claim 1 wherein said command recognition module includes: 

a microphone and a voice recognizer operatively connected to said microphone to: 
receive a voice signal, and 
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associate said voice signal with said at least one command in said command data 
set for controlling said at least one appliance; and 

a touchpad and a handwriting recognizer operatively connected to said touchpad to 
receive said handwritten signal. 

5 

5. The apparatus of claim 1 including: 

said command data set having a plurality of commands; and 

a main module operatively connected to said command recognition module and 
operative to communicate said command signal to said at least one appliance. 


6. The apparatus of claim 5 including: 

an appliance interface connected between said at least one appliance and said main 
module and adapted to communicate said command signal to said appliance. 

15 7. The apparatus of claim 6 including: 

a microphone and a voice recognizer operatively connected to said microphone to: 
receive a voice signal, and 

associate said voice signal with said at least one command in said command data 
set for controlling said at least one appliance; and 
20 a touchpad and a handwriting recognizer operatively connected to said touchpad to 

receive said handwritten signal. 

8. The apparatus of claim 7 wherein said command data set includes alphanumeric 


10 


characters. 
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9. The apparatus of claim 7 wherein said command data set includes symbolic 
characters. 

5 10. The apparatus of claim 7 wherein said at least one appliance is a cellular 
telephone and said command data set includes cellular telephone commands and 
alphanumeric characters. 

11. The apparatus of claim 7 wherein said at least one appliance is a personal 
1 0 computer and said command data set includes computer instructions and alphanumeric 

characters. 

12. The apparatus of claim 7 wherein said command data set is associated with a 
predetermined set of voice signals. 

15 

13. The apparatus of claim 7 wherein said command data set is associated with a 
predetermined set of handwritten signals. 

14. The apparatus of claim 7 wherein said command data set is associated with a 
20 predetermined set of handwritten and voice signals. 

1 5. The apparatus of claim 7 wherein said command data set is associated by a user 
with at least one voice signal. 
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16. The apparatus of claim 7 wherein said command data set is associated by a user 
with at least one hand signal. 

17. The apparatus of claim 7 wherein said command data set is associated with 
5 handwritten and voice signals modifiable by said user. 

1 8. The apparatus of claim 7 wherein said appliance is a car alarm. 

19. The apparatus of claim 7 wherein said device connects to a plurality of 
10 appliances and said main module and appliance interface cooperatively distribute said 

command signals to said appliances. 

20. A method of controlling an appliance within a vehicle comprising the steps of: 
providing a command recognition module having a command data set including a 

1 5 touchpad and a handwriting recognizer operatively connected to said touchpad; 
receiving handwritten signals; 

associating said handwritten signal with at least one command in said command 
data set for controlling said at least one appliance, and 

communicating a command signal associated with said command to said 
20 appliance. 
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